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ABSTRACT 



The focus of this paper is the Educational Resources 
Information Center's (ERIC's) products and information dissemination. Its 
purpose is to collect and review existing studies and analyze information: 

(1) about the effectiveness and appropriateness of the array of products ERIC 
now produces in meeting user demands and needs; (2) about the styles, format, 
and types of new products it will need to meet new customer demands; and (3) 
about the appropriate use of different media and technologies to disseminate 
information more effectively and in a more timely manner. Recommended 
methodological strategies that might be incorporated into a large-scale 
evaluation of the ERIC system are presented at the end of the paper, along 
with suggested key questions for the areas of product success, user needs, 
quality and standardization, and clearinghouses and partnerships. 
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EXECUTIVE SUMMARY 



The Educational Resources Information Center (ERIC) is the world’s largest and 
most frequently used education database. Through its mission, products, services and 
dissemination process, ERIC is the nation’s public resource to facilitate increased capacity 
building to address the imperatives of educational reform and Goals 2000. This fact 
notwithstanding, at the dawn of the twenty-first century, ERIC faces several dilemmas or 
challenges that must be effectively addressed if it is to continue in its role as a primary 
source of reliable and appropriate information about education. There are disparate bodies 
of data that suggest that, by-and-large, the ERIC system has been successful in providing 
products that meet users’ needs and demands. However, the system suffers from a paucity of 
systematic and comprehensive valuative data necessary to guide product maintenance, 
deletion and/or new product developments for the new millennium. To this end a 
comprehensive evaluation of the entire ERIC system is called for. 

This paper is part of a set of five papers commissioned by the U.S. Department of 
Education as preliminary preparation for a comprehensive system wide evaluation of ERIC in 
the future. The focus of the paper is ERIC’s products and information dissemination and its 
purpose is to collect and review existing studies and analyze information (1) about the 
effectiveness and appropriateness of the array of products ERIC now produces in meeting 
user demands and needs, (2) about the styles, format, and types of new products it will need 
to meet new customer demands, and (3) about the appropriate use of different media and 
technologies to disseminate information more effectively and in a more timely manner. 

Some of the key findings of the paper include the fact that although ERIC staff have 
made significant progress in developing and disseminating the kinds of products and services 
that effectively support users’ information needs and demands, it is also the case that old and 
new customers have rising expectations, driven, in large part, by new technologies. Thus, if 
ERIC is to maintain its competitive edge as a premiere educational information resource, it 
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must pay serious attention to the challenges and issues around new technological advances, 
such as issues of product obsolescence, timeliness, new product development, and customer 
satisfaction. 

As technology has changed over ERIC’s lifespan, the quality and reach of its products 
have been exemplary and continues to expand. ERIC has developed new initiatives (despite 
dwindling resources) and created new or additional products and services specifically 
designed for technologies such as e-mail and various Web tools. However, despite the widely 
acknowledged successes of these products and services, none of them are fully 
institutionalized within ERIC. This is the case primarily because of the tentative financial 
support for these products and services and the mechanisms for creating and maintaining 
other forward-looking products and services (Rising Expectations, 1996). A variety of data 
affirm that ERIC' s greatest internal asset is its people, staff, partners and contractors, and 
their abilities to be innovative and flexible. The decentralized structure and the availability of 
subject matter experts is the unique competitive edge ERIC has in place to meet the 
challenges for new product development and delivery mechanisms. The strengths of ERIC’S 
utilization of subject experts and its organizational partners are also repeatedly affirmed by 
the data. 

Finally, the paper recommends methodological strategies that might be incorporated 
into a large-scale evaluation of the ERIC system. The dynamic nature of technology 
absolutely suggests an evaluation model that utilizes both quantitative and qualitative 
approaches in a traditional scientific sense. However, models more related to product 
marketing research and information management (e g. data mining, trend studies, and focus 
groups) are also recommended. To ascertain how ERIC can best meet user demand and 
support public access to information and knowledge creation, some attention also must be 
given to an analysis of the competitive forces, private, profit and nonprofit, that are shaping 
the educational product development and dissemination landscape. 
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ERIC Products and Information Dissemination 

IvaE. Carruthers, Ph.D. 

INTRODUCTION AND STATEMENT OF PURPOSE 

We are currently in a historical period characterized by mandates for federal program 
accountability, national education reform, and increased competition for federal resources. It 
is also an era being transformed by the impact of an information technology revolution. 
Within this context, the National Library of Education’s Educational Resources Information 
Center - more commonly known as ERIC, the world's premier educational database research 
and materials support center is a national treasure. Through its mission, products, services 
and dissemination process, ERIC is the nation’s public resource to facilitate increased 
capacity building to address the imperatives of educational reform and Goals 2000. ERIC, by 
its very nature, is a critical national resource to foster the democratization of information and 
knowledge utilization in an age of technological privatization and the reality of a digital 
divide. ERIC’s presence and contribution to American education is arguably a centerpiece to 
the principles and foundation of American democracy in the twenty-first century. This fact 
notwithstanding, as the nation moves into the twenty-first century, ERIC faces several 
dilemmas or challenges as a twenty-first century information service, which must be 
effectively addressed if it is to continue in its role as & primary source of reliable and 
appropriate information about education. 

Despite its original mission, charged in the mid-1960' s, as an "educational research 
information center” and its success with “doing more” for users with “fewer resources," 
paradoxically, ERIC suffers from a paucity of systematic and substantial formal evaluation 
and assessment of its products and services. Former ERIC Director Keith Stubbs (1999a) 
noted that “ERIC has requested funds for a full-scale evaluation in each of its last three 
budget requests, but has not yet been granted additional funds for the purpose.” In FY 1999, 
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ERIC launched plans for a major system wide evaluation in FY 2000 by commissioning 
papers from five nationally known experts to frame the key issues and recommend evaluation 
methodologies in the key areas of: (1) Mission, Structure and Resources; (2) Database and 
Operational Processes; (3) Use of Technology in the ERIC System; (4) User Services; and 
(5) Products and Information Dissemination. This paper is in response to key area number 
five: Products and Information Dissemination. 

The purpose of this paper is to collect and review existing studies and analyze 
information : (1) about the effectiveness and appropriateness of the array of products ERIC 
now produces in meeting user demands and needs; (2) about the styles, format, and types of 
new products it will need to meet new customer demands; and (3) about the appropriate use 
of different media and technologies - electronic, print, microfiche, web sites, virtual libraries 
and so forth - to disseminate information more effectively and in a timely manner. Within 
this framework, this paper (1) identifies indicators of ERIC’s reach and successes; (2) 
synthesizes the disparate quantitative and qualitative data and observations about ERIC 
component products and dissemination process; (3) discusses critical issues that impact 
ERIC’s current and future ability to address product and delivery needs of its users; and (4) 
makes specific recommendations about the kinds of issues, questions and methodological 
approaches which might frame implementation of a comprehensive external ERIC program 
evaluation in the area of products and information dissemination. 

FRAMEWORK AND METHODOLOGY OF PAPER 

The methodological approach used to develop this paper included several research 
strategies. The author attended a 1999 ERIC national directors’ meeting that afforded an 
opportunity to listen to the nature of issues pending, as well as meet the parties most 
responsible for ERIC program implementation. Additionally, this meeting afforded an 
opportunity to hear first-hand some of the perceptions and challenges the Department of 
Education is addressing relative to Congressional and policy initiatives. 
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Existing studies and reports were collected and reviewed. These studies and reports 
included published evaluations, though limited, and a variety of published ERIC documents 
and internal reports. Sample publications and products from various ERIC Clearinghouses 
were gathered and reviewed. Moreover, several phone interviews were conducted with 
clearinghouse staff and institutional partners. Several ERIC reference librarians were also 
consulted. Discussions were held with representatives from several educational publishing 
companies and database consultants and selected educational and industry reports were 
reviewed. Lastly, a targeted survey was developed by the author and distributed to 
clearinghouse directors to provide an important perspective in this paper about the issues of a 
system wide evaluation going forward. Twelve responses, of sixteen requests, were received. 
In some cases, the responses reflected a collaborative response from the director and staff. A 
second survey developed by the author to tap user needs and concerns was posted on the 
Website of the ERIC Clearinghouse for Assessment and Evaluation. Unfortunately, the data 
received had little value in that while seven thousand (7,000) ERIC users were exposed to the 
survey, only thirty-nine (39) responses were noted, with only twenty-two (22) found to 
contain useable responses. 

In short, the basis of this paper relies much on a qualitative approach to data 
gathering and synthesis from a range of sources. It is the author’s feeling that the myriad 
sources of information, the scope of the information and the period over which the 
information spanned, adequately grounds the findings as reliable in order to achieve the 
assigned task of assisting ERIC in framing a plan for future system wide evaluation. 
EFFECTIVENESS AND APPROPRIATENESS OF CURRENT ERIC PRODUCTS IN 
MEETING USER NEEDS AND DEMANDS 

By all accounts, the ERIC system is the world’s largest and most frequently used 
education database with more than 3,000 locations around the world receiving ERIC print or 
database products. Approximately 900 of these maintain complete microfiche collections and 
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provide search services for clients. ERIC contains more than 950,000 bibliographic records 
of journal articles, research reports, curriculum and teaching guides, conference papers, and 
books. Each year approximately 33,000 new records are added. Originally, ERIC was a 
pioneering electronic bibliographic database and retrieval system used primarily by experts 
and professional researchers. However, over the past decade or more, the nature of the entire 
ERIC system’s audience has changed greatly to a more general audience of end-users 
accessing the system (Smarte, 1999; Stonehill, 1992). 

According to an ERIC Operations Task Force Report: Rising Expectations: A 
Framework for ERIC's Future in the National Library of Education, because of advances in 
CD-ROM technology and the Internet, by the mid-1990s, millions of people were using not 
only the ERIC database but also a wide variety of other ERIC resources - products and 
services - including peer-reviewed analyses and syntheses created specifically by ERIC for a 
diverse audience. Today, ERIC users include teachers, professors, and librarians; school and 
college administrators, counselors, instructional media staff, and support personnel; 
educational researchers, educational policymakers at every level; students and non-formal 
learners at every age and level, including adult learners; parents; health and social services 
personnel and caregivers who support families, parents, students, and children; and the media 
and business communities as they relate to education. In 1998, sixty percent of ERIC users 
were practitioners and the ERIC user base had surpassed one million contacts per year 
(Smarte, 1999). 

User Needs and Demands 

As reported in "Rising Expectations, ” ERIC users’ level of sophistication is 
increasing and, with the instant anytime, anywhere access that modem telecommunications 
provides, ERIC users’ expectations for information products and services have risen 
accordingly. These expectations include users being able to access ERIC products and 
services at home through their Internet-connected computers and television sets, having 
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access to data as soon as it is produced, and at little or no cost to them. In other words, users 

want all of the products and services of ERIC as quickly as possible, upon request, and as 

complete as possible, upon demand. Moreover, users are no longer satisfied with a 

bibliography or a list of citations. They increasingly demand full-text documents, custom 

syntheses, transparent referrals to other organizations and information sources, and real-time 

conversations with expert problem solvers. Responding to a 1999 Survey of Clearinghouse 

Directors conducted by the author (hereafter referred to as the Carruthers Survey— See 

Attachment 1), one ERIC Director said, 

Users are no longer content to be pointed in a direction toward 
information that may be of use, but rather they want to be sent 
copies of— or directed to Internet resources that summarize— the 
response to their question and package it so that they don’t have 
to do any primary research themselves (Carruthers Survey, 

Respondent 2). 

Most Directors responding to the Carruthers Survey agreed that the greatest user 

demands today are for information that is web-based and full-text or that is “digested and 

processed” so that users do not have to do anything but incorporate it into their project or 

assignment. Addressing this issue of user demands a Director observed: 

The current trend is to seek an answer to a particular question... 

Users in the past tended to be older college students or graduate 
students. Today, we are seeing younger users who ask for a 
completed final product. They do not usually ask for leads or 
research to be able to complete their work (Carruthers Survey, 

Respondent 3) 

These observations are consistent throughout studies and other forms of information about 
ERIC. ERIC users increasingly want to receive information in electronic format. Many of 
the ERIC Directors point to a trend of users ordering more items over the web and using e- 
mail as their primary vehicle for communication. Thus, the issues of technological change, 
full-text availability, and database currency are seen by many as crucial to the survival of the 
ERIC system. "Neither new nor traditional users will be satisfied with anything less than 
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this. There are too many other alternatives to ERIC available today" (Carruthers Survey, 
Respondent 5). 

In today’s world, people endeavor to become lifelong learners, improve education in 
their communities, and act with a greater understanding of educational issues. Although is 
has been difficult to capture with one hundred percent certainty, there is substantial 
demographic data on the thousands of users who send e-mail requests, visit ERIC exhibits at 
conferences, search the ERIC database, or obtain ERIC information via the Internet. 
According to the 1999 ERIC Annual Report, in 1998, ERIC Clearinghouses and components 
responded to 89,036 e-mail messages from users, 50,729 telephone calls, 34,733 letters, and 
6,480 visits. In this highly dynamic environment, ERIC must be prepared to meet the rising 
expectations of current and future users. In order to effectively and efficiently meet user 
needs and demands, the ERIC system will need to develop and implement a cycle of 
research, evaluation, and change in its products and services for educators (Rising 
Expectations, 1996; Stubbs, 1999b). 

ERIC Products: Meeting User Needs and Demands 

It stands to reason that hand-in-hand with reaching a widening and increasingly 
diverse audience of users is the need to provide the appropriate products as effectively and 
efficiently as possible that meet user needs and demands. The ERIC Clearinghouses create 
and produce a wide range of products in response to user needs and demands. 

Clearinghouses analyze and synthesize literature in their respective areas of expertise, create 
research reviews, bibliographies, state-of-the-art studies, interpretive studies of high-interest 
topics, digests, and a number of other publications that meet the information needs of the 
wide spectrum of ERIC users. More specifically, based upon a review of the document: All 
About ERIC (1 998), current ERIC publications and products by title and brief description 
include: 
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Clearinghouse Publications - Each year, the clearinghouses produce about 250 
publications, some aimed at specific audiences such as teachers and administrators, 
others are aimed at the traditional research audience. These products include flyers 
and brochures aimed at introducing the public to ERIC and the clearinghouse 
services, annotated bibliographies, books, newsletters, publication lists, search 
reprints, information bulletins, hot topics and/or occasional papers, and ERIC 
Information Analysis Product (IAP). 

ERIC Digests - Approximately 160 ERIC Digests are produced by clearinghouses 
each year. These two-page research syntheses provide brief overviews of current 
education issues. Significantly, Digest topics are determined by-and-large by the 
questions most frequently asked of the 16 subject-oriented ERIC Clearinghouses. 
Digests are often in a question-and-answer format and always include a list of 
additional resources. 

The ERIC Review - The ERIC Review is a free journal that reports on critical trends 
and issues in education as well as new ERIC developments. It is distributed to more 
than 25,000 subscribers and to targeted audiences interested in each issue’s particular 
topic or theme. Copies are also distributed at education-related conferences and 
meetings and are mailed to individuals and groups that request them. Past issues of 
the Review have focused topics such as school-to-work transition, K-12 computer 
networking, information dissemination, inclusive schools, and the path to college. 

Parent Brochures - In keeping pace for demands from parents and other interested 
users for information on the role of parents in their children’s education, ACCESS 
ERIC, in collaboration with ERIC Clearinghouses and their networks of experts, 
produce this series of “easy-to-read” brochures for parents. Each year a list of new 
Parent Brochure titles are sent to more than 9,000 parents, parent educators, 
counselors, and other interested individuals. Parent Brochures always include a list 
of organizations and publications that can provide the user with further information 
on the topics. . 

Products About ERIC - In addition to All About ERIC, ERIC produces an array of 
products designed to introduce individuals and organizations to the resources of the 
ERIC system and to assist them in their search for relevant and usable education 
information. These products include: 

A Pocket Guide to Eric - a pocket-sized booklet that provides a brief 
description of the ERIC system and its products and services. 

ERIC User ’s Interchange - a newsletter for librarians and ERIC resource 
collection providers that is published semiannually and covers information on 
new ERIC products and services. 

ERIC In Action - a 1 5-minute video that describes the basic structure of 
the ERIC system, major benefits the system offers to users, key features of 
the ERIC database, and information about how to locate materials of interest. 
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ERIC Transparencies - consists of a set of 48-paper transparency masters 
that includes information on what ERIC Clearinghouses do, how documents 
are selected for the ERIC database, search strategy development, AskERIC, 
online ERIC document delivery, and many other topics. 

In addition to these products, the ERIC Processing and Reference Facility 
produces a number of system wide support products such as processing manuals, 
bibliographies, indexes, directories and Ready References, a brief handout that contains 
ERIC database-related information that users need. Other publications include: The 
Educational Resources Information Center (ERIC): An Annotated Bibliography of 
Documents and Journal Articles About ERIC, the ERIC Title Index, and the ERIC Identifier 
Authority List. Further, ACCESS ERIC produces several reference and referral publications 
which are accessible online. These products include: the Catalog of ERIC Clearinghouse 
Publications, a guide to more than 1,400 free or low-cost education titles published by ERIC 
Clearinghouses; Directory of ERIC Resource Collections, a directory of over 1,000 libraries 
and other organizations that offer access to the ERIC database, microfiche, and ERIC 
publications; ERIC Directory of Education-Related Information Centers, a directory which 
describes 450 resource centers and agencies that provide information or assistance in 
education-related areas; and the ERIC Calendar of Education-Related Conferences, a guide 
to over 525 international, national, and regional education conferences (All About ERIC, 
1998). 

ERIC Products: Human Dynamics 

It is clear that ERIC staff are committed to responding to user needs and demands as 
effectively as possible and that they engage in a wide range of activities dedicated to this end. 
In fact, the dedication, commitment, and expertise of ERIC Clearinghouse staff are cited 
repeatedly as among ERIC’s greatest strengths. One respondent in the Carruthers Survey 
noted that “the different clearinghouses work with wonderful and talented people who trust 
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ERIC’s reputation and in turn produce excellent Digests, major publications, and other 
products” (Carruthers Survey, Respondent 3). 

In 1998, ERIC Clearinghouses produced more than 660 information products. Of 
note is the collaborative ways in which these products are produced, the extensive networks 
through which they are distributed, and the wide attention they receive and appeal they have 
among users. A number of ERIC publications are produced in partnership between ERIC 
Clearinghouses and academic institutions, professional associations, private publishers, and 
other organizations (Smarte, 1999). Never content to rest on their past reputation and 
successes, ERIC staff constantly work together to come up with new ways to respond to 
current and potential user information wants and needs. In this context, the many special 
projects that the system proposes to the ERIC Program office (e.g. AskERIC, National Parent 
Information Network) often end up being ERIC’s greatest success stories and they manage to 
keep ERIC evolving in response to changing user requirements and expectations (Carruthers 
Survey, Respondent 2). 

Moreover, ERIC’s increased use of electronic formats to disseminate its information 

o 

is an example of the system’s ongoing efforts to provide appropriate mechanisms through 
which users can access ERIC products and subject matter experts. ERIC is making excellent 
use of the Internet to answer questions and disseminate its data. The Web allows ERIC staff 
to disseminate information quickly and very efficiently to a great number of people. In this 
context, a respondent in the Carruthers Survey points to ERIC’s Web presence as one of its 
greatest strengths: 

One of its greatest strengths. ..obviously disseminating via the 
Internet as one look at the statistics will show. ERIC use is 
at an all-time high and this directly correlates to Internet access 
to full-text ERIC products and free database searching... A 
forthcoming book on the second fifteen years of ERIC, being 
published by EDRS, will suggest that much of the ERIC 
system feels that synthesized products are also helping to make 
ERIC more widely recognized and respected in the education 
field. Products such as ERIC Digests, Trends and Issues 
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Papers, and Parent Brochures are being produced in response 
to user requests for synthesized information (Carruthers Survey, 
Respondent 2). 



There is near consensus among ERIC staff that the system’s Internet-centric strategies 

strengthen considerably its capacity to effectively address user needs and demands and in a 

more timely manner. As an ERIC staff member noted: 

They strengthen ERIC’s response to its user community by 
providing access to collections of resources that are currently 
unavailable to many educators because of cost, location, and 
technology restrictions. Clearinghouse Web sites provide links 
to primary sources of state, national, and international information. 

The sources can be as diverse as original handwritten texts of 
speeches made by national and international leaders to drawings 
of historic buildings and locations, to great works of art and 
musical scores of renowned composers (Carruthers Survey, 

Respondent 5). 



Some staff note that ERIC’s use of the Web enhances its capacity to reach and provide 
services and products to people who otherwise would not have access to ERIC (e.g. persons 
with special needs, rural, remote locations and/or foreign). ERIC has always been on the 
leading edge in using the Internet for information dissemination, however, some ERIC staff 
have noted that Internet access, in and of itself, does not differentiate ERIC or keep it more 
viable. Rather it simply puts the system on the same level as other services and industries. 

Finally, an important ERIC function is the development of outreach activities that are 
designed to attract new users to ERIC and to update experienced users on the latest ERIC 
products and services. These outreach efforts include exhibits and presentations at 
education-related conferences. For example, in 1998, ERIC staff members participated in 
more than 470 conferences, meetings, and workshops; gave more than 230 presentations; and 
staffed exhibits at more than 80 major conferences (Smarte, 1999). On the road, ERIC staff 
solicits feedback on various ERIC products that are, in turn, incorporated into continuous 
efforts to update and/or redesign products to meet user needs and demands. 
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ERIC Products: User Satisfaction 



Although the need for a full-scale, system wide and independent assessment of ERIC 
products and services as well as user need, demands and satisfaction continues to be 
paramount, we can nonetheless, learn much about the effectiveness and appropriateness of 
ERIC products in meeting user needs and demands from the feedback ERIC staff members 
receive on the road and from Customer User Surveys. Customer feedback and survey results 
have driven important changes, large and small, in clearinghouse operations and have set the 
stage for a system wide evaluation of ERIC, the results of which are expected to significantly 
guide ERIC’s work in the new millennium (Smarte, 1999). 

As the U.S. Department of Education and other federal agencies look closer and 
closer at accountability in federal programs an increased emphasis has been placed on 
evaluating the impact of programs such as ERIC. Beginning in January 1999, new five-year 
contracts required each ERIC Clearinghouse to regularly gather data/feedback regarding 
customer satisfaction with its products, services, and how its work contributes to improving 
education. In order to meet this requirement, the clearinghouses will conduct annual surveys 
using a customer satisfaction survey instrument that was approved by the Office of 
Management and Budget (OMB) for use by all National Library of Education Programs 
(NLE). 

Based on the data reviewed for this paper, it is clear that ERIC staff have always 
worked hard to meet user needs and demands and, over the years, they have made significant 
progress in developing and disseminating the kinds of products and services that effectively 
support users’ information needs and demands. In this regard, ERIC has an outstanding track 
record for customer satisfaction. Each year ERIC receives hundreds of thank-you messages 
that provide anecdotal evidence of ERIC’s usefulness to a variety of education stakeholders. 
Historically, ERIC components have also used formal approaches to gather information on 
who uses ERIC, how they use it, and their level of satisfaction with ERIC products and 
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services (Smarte, 1999). However, a caveat is necessary here. It should be noted that 
clearinghouses have generally not been able to measure the benefit of their products and 
services in any quantitative or comprehensive manner. The anecdotal data is usually from 
those users who are well pleased with the products. Additionally, survey data, to date, 
generally does not get at all of the key questions necessary to a systematic assessment of the 
overall system. 

The aforementioned caveat notwithstanding, the results of various user surveys 
conducted during the 1990s indicate that customer satisfaction with various ERIC products is 
quite high. For example, a 1994 survey reflected a 95 percent rate of user satisfaction with 
the speed of response; an 88 percent satisfaction rate with the quality of response; and 97 
percent indicated that they would recommend the service. These findings are consistent with 
the results of various surveys conducted in 1998. An analysis of 600 questions received from 
parents in response to a Newsweek article featuring the National Parent Information Network 
(NPIN)— a special project of the ERIC Clearinghouse on Elementary and Early Childhood 
Education (ERIC/PS) and the ERIC Clearinghouse on Urban Education— conducted by 
ERIC/PS found that of the parents surveyed, most were pleased with the service NPIN 
provides and 89 percent of these parents rated the materials they received as useful, 
particularly those written specifically for them. In addition, more than 70 percent indicated 
that they felt more comfortable in their role as parents as a result of the information they 
received and more than 65 percent indicated that NPIN saved them “time and effort in 
obtaining information." In 1997, the NPIN site was visited in excess of one and one-half 
million times, which is a testament to its popularity and usefulness (Smarte, 1999). 

Likewise, there seems to be a general consensus about the popularity and usefulness 
of AskERIC. Although the AskERIC Q & A service is not a product per se, the 
comprehensive responses provided by AskERIC staff to individual users is certainly one of 
ERIC’S great strengths in terms of giving ERIC worldwide recognition and visibility 
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(Carruthers Survey, Respondent 1). Since its appearance in 1992, AskERIC has greatly 
increased the visibility of the system. In 1998, an AskERIC user survey found that 92 
percent of the 196 users (a 31 percent response rate) who responded to questions about the 
value and quality of AskERIC reported that the information they received was relevant to 
their question and 93 percent said that they would recommend the service to someone else. 
The respondents also indicated that AskERIC had saved them either time (91 percent) or 
money (48 percent) (Smarte, 1999). 

Other ERIC products as well are popular and receive high praise from users. For 
example, the Parent Brochures are highly popular with the public as are the ERIC Digests. 
Both publications are good examples of ERIC’s ongoing efforts to provide appropriate 
products to its public— products that users need and demand. Given the high demand for 
synthesized information, the ERIC Digests have grown continuously in terms of demand for 
and use. Users of the ERIC Digests typically say that they think very highly of the Digests 
and use them frequently. One staff member is quoted in an ERIC History Manuscript as 
saying that “the ERIC Digests are by far the most popular item of the system. People know 
(the) Digests and they love them.. ..It’s made information more accessible to more people" 
(quoted in Colker, 1999). The ERIC Review, which debuted in 1990, is often cited for its 
quality. The large and growing number of subscribers provides some indication that it is 
meeting user needs fairly effectively. In 1998 alone, nearly 1,000 new subscribers signed up 
to receive a copy of the ERIC Review. Along these lines, a survey of readers of NCLEnotes, 
the newsletter of the Adjunct Clearinghouse for ESL Literacy Education, revealed that an 
average of 6 people read each copy of the newsletter, thus a distribution of 6,500 copies 
actually reaches approximately 39,000 people. 

From an internal perspective, ERIC is a network of educational related resources, 
people, products and services, delivered by a decentralized infrastructure consisting of a 
combination of various subject matter experts and technology support. To the customer, 
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however, ERIC is more of an infinite repository of information, including databases, free or 

low-cost products and publications, information retrieval, referral services, and subject- 

matter experts. Although surveys, anecdotal and other data show that ERIC users are 

satisfied by-and-large with the products that the system offers, it is clear that ERIC has not 

yet satisfied customers’ needs for full-text delivery. For example, in a survey conducted by 

the ERIC Clearinghouse on Elementary and Early Childhood Education (1999), a typical 

sentiment among users is expressed in the following quote: 

Any changes you would make to include more full-text 
journal articles would be a great improvement...students 
appreciate being able to leave the computer with their 
articles in hand, rather than having to check our online 
catalog and then check the shelves and then photocopy 
or check out heavy bound volumes of periodicals from 
the library (quoted in Rothenberg, 1999, p. 6). 

The majority of the data reviewed for this paper also shows quite clearly that most 
users are not content with the system’s present incapacity to provide comprehensive coverage 
of education-related materials (in essence, to be a national catalog of all key education 
resources in all formats) and would like to see this function greatly expanded. In general 
terms, although based on small sample responses, two primary areas of customer concern 
identified by ERIC staff are (1) the content of the database and, (2) the availability of the 
items described in the database (See for example, Rothenberg, 1999). 

An in-depth analysis of an array of internal documents distributed among and 
between ERIC staff, show quite clearly that staff are consistent and unified in their belief that 
the system must embrace and promote a cycle of evaluation that effectively taps into user 
current and changing needs and concerns as well as their overall satisfaction with ERIC 
products. The potential for ERIC to document its successes, user satisfaction with its 
products and services, maintain current users and reach new audiences, will require a more 
relational posture with its users. Motivating users to participate in product assessment and 
evaluation is essential but not an easy task. By all accounts, however, ERIC is committed to 
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building for the future based on user input. ERIC staff has always engaged its customers in 
this process and they have effectively used this input to fashion and/or redesign various 
products consistent with users’ needs and concerns. This process of soliciting and 
incorporating user input into the process of product development, maintenance, and redesign, 
will surely continue in an even more formal and systematic manner, given that 
clearinghouses are now required to conduct an annual survey of users regarding their 
satisfaction with and use of various ERIC products and services. In this regard, some ERIC 
staff have suggested that their “evaluation efforts need to be as directed towards monitoring 
how we are doing as we go along as to summative evaluation where we examine what we 
have or have not accomplished.” (Topics for Discussion, 1999: p. 1). 

GOING FORWARD: CHALLENGES AND ISSUES 

A major industry paradigm shift in marketing and providing most educational 
products and services has taken place. Product appropriateness and effectiveness are 
increasingly being determined by both the quality of services and their delivery. And 
increasingly, services and service delivery are construed as products. A seamless continuum 
of product options, content, format and type, is being driven by technological access, creating 
increased demand for Web-based products and services. Over the past 30 plus years, ERIC 
has demonstrated unmatched capacity to persevere and flourish as an international resource, 
with an increasingly diverse user community, a myriad of products and multiple points of 
dissemination. To meet future user needs and demands in a growing and increasingly 
competitive environment staff will need to continue to be as resilient. As ERIC Directors and 
staff move forward in the new millennium and consider what need to be done to sustain and 
increase ERIC’s capacities and capabilities in order to remain a “world class act,” activities 
must include serious attention to the challenges and issues around new technological 
advances, issues of product obsolescence, timeliness, new product development, and 
customer satisfaction. 
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Managing Product Obsolescence and Timeliness 

Clearly, there is a need for ERIC to manage product change and obsolescence. 
However, phasing out obsolete products and mechanisms for the dissemination of 
information in a timely manner will be a challenge. As far back as 1987, Distinguished 
Professor of Higher Education, Kenneth Tollett cautioned that “any significant change in the 
structure, organization, or operation of ERIC should be approached with great 
circumspection and care." (quoted in Colker, 1999: p. 3). 

As our earlier discussion suggests, the styles, format, and types of most of ERIC's 

current products are generally viewed by ERIC staff and its users as effective and 

appropriate. However, as the world of technology and telecommunications continue to 

change, so too must the style, format, and types of some of ERIC’s products as well as its use 

of different media technologies in order to meet new customer demands. In response to a 

question in the Carruthers Survey that asked if information was as accessible from ERIC as it 

could be, several respondents pointed out that not all information in the ERIC database is as 

accessible as it could be. These respondents firmly believe that everything ERIC has on 

microfilm or microfiche should be available on the Internet as well. The obsolescence of 

microfiche as a storage medium is an issue that crops up time and again in ERIC staff and 

users’ observations and comments. Most sources that address the future of ERIC view 

microfiche not only as outdated but also as difficult to use. In the Carruthers Survey, several 

Clearinghouse Directors identified microfiche as one of ERIC's greatest weaknesses. For 

example, according to one Director: 

It locks us not only into a place-based information world, 
but also a print-based world... We are a distributed network 
of expertise that has to fit into a one-place bottleneck (the 
Facility) for editing and entry...The clearinghouse structure 
And the Internet are the strong parts of ERIC... the future. 

The Facility and microfiche are our 60's legacy. They are 
Quickly becoming the chains that bind our progress 
(Carruthers Survey, Respondent 1 1). 
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Another key issue that is repeatedly described as a problem by both ERIC staff and 
its users is timeliness — the delay in processing CUE (journal literature) records. The current 
processing method results in, at least, a six-month delay before the records appear in the 
ERIC database. By the time new records appear in the database many of them are already 
out-of-date. This delay in database processing is added to what some see as an already 
extremely late (slow) database processing timetable. In suggesting how to improve the 
efficiency and speed with which materials are prepared for and entered into the ERIC 
database and made available to ERIC users, a work flow analysis is recommeded as a system 
wide strategy. This recommendation addresses a repeated concern expressed by ERIC staff 
that staff reorganization vis-a-vis technology deployment at the Facility may be required (see, 
for example. Input on Draft Questions..., 1999). 

Some sources speaking about the issue of timeliness note that delays can also be 
related to the need to obtain releases for documents. Although the reproduction release is 
online, some ERIC staffbelieve that some form of electronic signature approval or 
authorization would help speed up the process. (Input on Draft Questions..., 1999, 
Respondent A). And, in some cases, delays can also be seen in the fact that materials 
contained in the World Wide Web database are a month behind the quarterly CD-ROMs. 

One respondent providing input to "Draft Questions about the ERIC Facility Recompete " 
questioned whether there is a possibility that either implicit or explicit arrangements give 
vendors an advantage (Input on Draft Questions..., 1999, Respondent C). Finally, some 
observers suggest that timeliness is both a virtue and a weakness of ERIC. While ERIC 
Clearinghouse staff are very current (timely) on their subject area and current educational 
issues generally, the database itself is not current enough (slow). 
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Managing Technology Change and New Product Development 

Managing obsolescence and timeliness is directly linked to new product development 

procedures. The following three cases exemplify this point. 

In 1987, an ERIC partnership with Silver Platter led to the first 
CD-ROM vendor partnership. Later DIALOG and OCLC would 
join ERIC as CD-ROM vendors. In 1995, ERIC made arrangements 
with the National Information Services Corporation (NISC) to buy 
and distribute their product at cost to the educational community. 

The ERIC CD-ROM collection is now widely distributed. However, 

CD-ROM technology is becoming obsolete among both the user and 
producer communities. 

The UMI Full Text Pilot Project of 1991 was a different story. In 
this pilot, a dedicated workstation was required to access a full 
text CD-ROM. The result was that the cost was prohibitive to 
the targeted market. This project did not succeed. According to 
Bob Stonehill: “. . . we had the right idea, it was just a decade 
ahead..” 

The more recent failed relationship between America OnLine 
(AOL) and ACCESS ERIC may be more reflective of AOL' s 
lack of vision about the education market and/or its internal 
capacity. However, an observation about this failed partnership 
may be that it highlights problem areas that could impact future 
relationships between public and private partners. Such issues 
as interoperability, copyright and royalty compensations, 
leadership and corporate ownership changes are all implicit in 
the ACCESS ERIC AOL venture (Colker, 1999, p. 138). 



These examples of joint venture with the private sector underscore the necessity for 
managing the technology curve. Some of it is luck; but most of it is having a knowledge of 
your user community’s needs and capacities, where the technology industry is going, and 
making the decision o f where to put limited resources with what anticipated gain. 

There is little doubt that new technologies, particularly the Internet, have had a 
dramatic impact on the both the expectations and needs of ERIC users. ERIC has been a 
leader in using the Internet for information dissemination and should be proud of its 
leadership role on the Internet as other organizations have noted (Lankes, 1999). As a history 
of ERIC reveals, it has always been able to adapt its product mix to technological changes: 
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from film to microfiche to OCR to CD-ROM to internet-centric and some full-text delivery. 
Thus; it is very important to focus on technology as a delivery system and not as a product. 

ERIC’s history of leadership and current role in utilizing technologies to deliver/ 
disseminate its products notwithstanding, survey data indicate that many of its customers are 
not content with the system’s current-level use of the Internet. This holds true for various 
ERIC staff as well. Many ERIC stakeholders indicate that in shaping the future of ERIC 
products, the system must move from a paper to an electronic world perspective. For 
example, according to Richard Stroup (1999), ERIC’s current perspective is overwhelmingly 
and inappropriately paper oriented. In the future ERIC must move to Web-based versions of 
several of its products such as the Source Index, the ERIC Processing Manual, the ERIC 
Thesaurus, the Identifier List, the ERIC Acquisitions Arrangements List, and the Title Index. 
Most ERIC staff and others seem to agree with Stroup's observations indicating that the paper 
versions of these products are bulky, expensive, and as Stroup observed: “horribly inefficient 
to use." (p. 1). 

In general, ERIC staff believes that continuing with print-based activities results in a 

limited database which reduces the usefulness of the ERIC system for users. Not all 

stakeholders agree, however, that the answer is to get rid of all print resources. For example, 

while acknowledging that the mass of education information no longer consists primarily of 

journal articles and printed reports, a customer quoted in a 1999 report on Responses to the 

Rising Expectations ” Report, said: 

While it is clear that there are loads of other no-print 
resources available to the education information consumer, 
print resources continue to provide the primary reputable, 
peer-reviewed source of research information on education 
issues. This focus is essential if ERIC is to continue to 
maintain its importance to those of us who need access to 
research-based information (quoted in Rothenberg, 1999, p.3). 
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Among those users who provide input to ERIC staff regarding their information 

needs, almost universally these customers want products such as full-text articles on the Web. 

Most users would like to have such products at no cost to them, although some them indicate 

that they would not mind paying a small fee. Furthermore, in response to a question that 

asked if ERIC was as accessible to users as it could be, several participants in the Carruthers 

Survey felt that it was not as accessible as it could be precisely because full-text documents 

are not easily accessible through the Internet. Several respondents related the issue of access 

to the need for clearinghouse standardization, one of whom articulated it as follows: 

Access to the ERIC database needs standardization. Currently, 

There are at least 3 CD-ROM producers, each with a unique 
Record format, with unique field names, and with unique search 
parameters. In addition, clearinghouses have mounted the data- 
base on their Web sites, again each using different search engines. 

This variety of interfaces results in a confusing number of record 
structures and field names. Another access problem occurs when 
specific information is easily retrieved using one version of the 
ERIC database while using another version makes retrieval of 
the same information impossible or extremely difficult. Also, 
some Web site versions of the database provide access to current 
records. Other Web sites provide access to the entire database. 

(Carruthers Survey, Respondent 5). 



These issues notwithstanding, decisions about product obsolescence and new product 
development must consider the latent audience that may or may not have access to the 
technology. The other side of putting everything on the Internet is the issue of the “digital 
divide. ” Not everyone has access to the Internet. Even as ERIC responds to new 
technologies and develops new products within the context of these technologies, it must 
make sure that its information continues to be accessible to as broad a user audience as 
possible, including those people who do not have access to the Internet. One ERIC staff 
member put it very succinctly: 

As long as there continue to be low-tech schools and teachers 
without Internet access on a daily basis whether from work or 
home, ERIC must continue to offer other means of accessing 
its information than just the Internet. However, for those who 
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do have Internet access, ERIC has the potential to expand its 
audience base. Of course, this continued audience expansion 
via the Internet creates a dilemma The more people that 
access ERIC without additional funding available to ERIC 
components to beef up their Internet sites and keep them 
current, the more people may not be pleased with the level 
of products or services they receive (Carruthers Survey, 
Respondent 2). 



A number of other issues related to technology and the direction for new product 
development can be culled from the data Some of the most frequently cited include: 

• ERIC must continue to support and not compromise its products and services 
to people with special needs 

• Provide more products, in more formats, that synthesize, interpret, and add 
value to existing resources. 

• Give greater attention to teaching and tutorials to provide users of ERIC with 
searching and analysis skills rather than information and answers. 

• Maintain a high level of products and services as well as expand and develop new 
products. 

• ERIC is impaired by a low-level of funding. 

• ERIC is impaired by limited resources and an aging operations framework. 

• Provide delivery of digital products (e.g. software, images, data sets). 

• Provide organizational schema, descriptions of Web sites related to education. 

• Develop/enter into collaborative relationships to assure easy journal article 
delivery. 

Responses to ERIC customer surveys also reveal the need for more user training, especially 
to help people get the best search results from the ERIC database and understand how to use 
the results. For example in 1998, researchers concluded that ERIC users are conducting poor 
searches and are accepting one or two somewhat relevant hits rather than conducting 
thorough searches to locate the best citations. ERIC is already addressing this issue by 
adding more "smart " features to its database search engines and by expanding online search 
tutorials. These and other data from users inform staff as to whether or not current products 
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produced by ERIC are the right ones; how well they work for users and what their specific 
benefits are; and which products should be discontinued and which are needed in the future. 
These insights are vital to future product development activities as they provide clear 
direction for the “new” ERIC— an ERIC that can meet the needs of users in the twenty-first 
century. 

In sum, what emerges from most of the data about ERIC products and services is 
that as technology has changed over ERIC’s lifespan, the quality and reach of its work 
products has been exemplary and continues to expand. ERIC has developed new initiatives 
(despite dwindling resources) and created new or additional products and services 
specifically designed for technologies such as e-mail and various Web tools. These include: 
AskERIC, NPIN, ERIC Test Locator and related systems, low-cost CD-ROMs, full-text 
digitization of ERIC documents, and ERIC's “smart” search engines on the Web. However, 
despite the widely acknowledged successes of these products and services, none of them are 
fully institutionalized within ERIC. This is the case primarily because of the tentative 
financial support for these products and services and the mechanisms for creating and 
maintaining other forward-looking products and services (Rising Expectations, 1996). 

While there is still much work to be done in terms of keeping up with customers’ 
rising expectations, particularly as they are linked to the issue of the Internet, in this era of 
extended collaboration, knowledge and resource sharing among the various components of 
ERIC must continue. This process of collaboration must impact all production and 
distribution tasks. Moreover, a delicate balance and sensitivity to ERIC as both an archival 
and living database must be maintained. In planning the demise of old products and the 
development of new ones, the recognition that ERIC can both help to close the gap or 
contribute to the digital divide is critical. 
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Funding 

Funding has always been an issue for ERIC. Authors of the "Rising Expectations" 
Report point out that one of the two major dilemmas that compromise ERIC’s continued 
ability to deliver what customers expect is its history of dwindling resources, particularly 
funding. For example, while per-pupil expenditures have doubled and federal education 
expenditures have quadrupled (in constant dollars) over the last 30 plus years, ERIC’s 
resources have been cut by 80 percent. The paradox here is that over the years, ERIC has 
become excellent by doing more with less. The system has a record of remarkable efficiency 
despite the lack of appropriate funding and other resources, leveraging at least ten times its 
own resources. The various clearinghouses, such as the Clearinghouse on Educational 
Management, have long labored to use their limited resources to maintain existing services 
and also go where the majority of their users have gone: the Internet (Smith, 2000). 

However, as the “Rising Expectations" Report suggests, such success could also be ERIC’s 
undoing because without substantial increases in resources, staff will not be able to respond 
to the rising expectations of customers at their high level of performance of the past. Bottom 
line, in the future, ERIC will not be able to continue its record of doing more with less. In 
fact, the ERIC Operations Task Force predicts that ERIC will indeed do less and less well 
with level funding. 

The issue of funding is a major concern among Directors and staff across all 

sixteen ERIC clearinghouses. When asked in the Carruthers Survey to identify ERIC’s 

greatest weaknesses and threats, respondents overwhelmingly cited "limited funding’’ “low- 

level funding ” “insufficient funding ” 'ho targeted funding for research and development or 

new systems. " The remarks of one respondent is typical of those made by most respondents 

relative to the issue of funding and ERIC’s future: 

Without increased funding, ERIC cannot serve the next 
generation of educators, parents, policy makers and 
students in the manner that fits the expectations of those 
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users and in the tradition of ERIC. (Carruthers Survey, Respondent 5). 



The general consensus is that limited or no growth in federal funding makes it extremely 
difficult to improve the system and in many ways, suggests limited support from the 
Department of Education. In this context, active and appropriate advocacy for ERIC within 
the Department of Education and Congress is suggested. 

Human Resource Utilization Through Partnering 

Many stakeholders suggest that ERIC’s longevity has been due specifically to its 
ability to attract and sustain some 650+ mutually beneficial private partnerships, resulting in 
innovative ways to leverage funds and in-kind resources. The role of university and 
professional organizations cannot be minimized in this regard. Likewise, long-standing 
relationships with EDRS, Oryx press, and Dialog are a few examples of such relationships 
involving the corporate sector. Finding common ground around issues of exchange, i.e. 
intellectual property and costs for development and dissemination has not been easy. "Of 
the $136M researchers estimated that it cost to operate the ERIC system, only $5.6M was 
underwritten by the federal government” (Colker, 1999, p. 36). This kind of return on 
investment should not go unnoticed. This strength of leveraging resources, in conformity 
with appropriate technical advances and interfaces, may define the new ERIC products. 

In an interesting discussion of Adjunct, Affiliates and Partners: Building an ERIC 
Network for the 21 st Century, R. David Lankes (1999), Director of ERIC Clearinghouse on 
Information and Technology, suggests that recent events have prompted ERIC to question 
how the system interfaces with partners and other organizations in general. According to 
Lankes, currently, there does not appear to be a shared definition of partners and adjunct. 
Added to this problem is the advance of the Internet that has had the tendency to blur 
traditional scope boundaries. Lankes looks at new partners and redefines existing ones in 
terms of what he views as the “evolving ERIC database. In brief, he believes that ERIC’s 
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future is through partnerships; partners are essential to ERIC better utilizing its limited 

resources. In this context he concludes: 

ERIC’s mission and audience have outgrown its government 
supplied resources and it will only get worse regardless of 
administration... In order... to continue to excel, it must build on 
its greatest strength, distribution. ERIC must extend its concept 
of itself as a virtual organization and create strong linkages with 
strong partners. . . In this way, ERIC cannot only provide more and 
better services to its users, it wall become an essential part of more 
and more organizations... ERIC has the potential to lead in extending 
its influence through partnership. We are already headed in this direction, 
but we must do so continuously and cautiously. (Lankes, 1999, p.7) 

Lankes discussion highlights how key a consideration of partnerships is in any evaluation of 

ERIC, whether system wide or clearinghouse specific. 

Issues of Copyright and Intellectual Properties 

ERIC clearinghouses have been able to adhere to copyright and fee guidelines 
however, they do not control these guidelines and today’s environment poses some new 
challenges. In this marketplace, with the privatization of and profiteering from both the 
Internet and education growing at a fast pace, issues of ownership, supply and demand, 
present different constraints and guidelines for cataloguing and utilizing information. Parties 
with interest in the library, computer, and publishing markets potentially intersect with 
ERIC’s mission. By way of example, they include Educational Research Services, 
NetSchools, and Scholastica. Thus, the way in which ERIC incorporates policies and 
technologies to document, link, and/or interface to other properties/products must be 
carefully considered, particularly as ERIC begins to consider the incorporation of video and 
multi-media materials into the scope of its products. 

The multimedia and Internet-centric environment of document production and 
dissemination have created a highly dynamic environment over issues of copyright and 
intellectual properties. ERIC, as the largest educational database in the world, in a time when 
industry is reawakening to the potential profitability of educational and multi-media materials 
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development, will have to chart new territory. The U.S. Advisory Council on the use of 

National Information Infrastructure cautions: 

The use of the National Information Infrastructure poses 
new legal and practical challenges in balancing the interests 
of owners and users of copyrighted works in a changing 
technological environment, ensuring an adequate level of 
protection to encourage owners to make their works available 
on the NH while permitting appropriate access to those works. 

(U.S. Advisory Council on Nil, p. 12) 



Without question, the issues of control and compensation over intellectual and 

creative properties, in a global network environment, bring new players with new interests to 

the environment. The edutainment/information corporate mega mergers can signal 

possibilities for partnership and/or possibilities for displacement. Recognizing the 

legislative limitations upon clearinghouses to assess royalty and fee charges, Stonehill 

indicated that ERIC was pursuing options by which reasonable usage fees can be collected 

from online vendors (Stonehill, 1992: p. 5). According to Nelson Heller (1993): 

For publishers of on-line databased and other electronic 
reference products, the Internet represents both a threat 
and an opportunity The threat: copied files conceivably 
can be distributed to thousands of computers around the 
world in minutes...accessed by any user. The opportunity: 
the Internet is gradually being converted from a “free 
government subsidized service to a fee-based commercial 
service, creating a new distribution channeL.this has 
special significance for education...(Heller, p 4) 



The reality is that more and more conference presentations will be utilizing 

technology where only salient points are spoken to using presentation applications and, in 

some cases, where just in time delivery, including translation documents are available at the 

conference site. Additionally, as one respondent to the Frameworks Survey noted: 

Many organizations, including ASHA, depend on revenues from 
the sale of publications, including journals and journal reprints. 

Having those available free to people who are not members of our 
organization, when our members do not get such free would have 
serious financial consequences (quoted in Rothenberg, p. 6). 
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In setting out new goals for OCLC, K. Wayne Smith, President, noted that the 
addition of educational services for the library and education communities is a priority. 
“Technological advances are bringing both new opportunities and problems. The old 
paradigms are giving way to new ones. But with all this change membership values of 
cooperation, sharing and working together for the public good clearly remains more vital and 
compelling than ever. (Strategic Plan Letter, 1997). 

Thus, in this new environment, a partner can easily and concurrently be a 

competitor. The implication for ERIC’s future funding and leveraging of its human and fiscal 

resources is at the heart of effectively allocating resources to meet user demands. In raising 

the question of ERIC’s future role in indexing, Stonehill (1992) highlights one of the areas 

that ERIC must look at in terms of being able to best leverage its human resources. Phyllis 

Steckler at Oryz Press suggest that abstracting and indexing can now be done electronically 

and resources for those functions deployed to dissemination. (Colker, p. 185) The views of 

Smith and Steckler point to some major envisioning of how twenty-first century ERIC 

staffers spend time and dollars. Without diligent and proactive marketing of ERIC to old 

supporters and new audiences, Director Larry Rudner’s observation is telling. According to 

Rudner, who is a strong voice to Congressional Staffers on behalf of ERIC: 

While we are extremely well known in education and 
library science, we are virtually unknown outside of 
those fields. We don' t make headlines.. we are service 
people at the grass roots level. If we don' t raise aware- 
ness there is a good possibility of our being killed without, 

Congress even knowing that they are killing us. 

(quoted in Colker, p. 140) 



ERIC: Perception vs. Reality 

Rudner’s observation raises the question of perception, perception, perception. The 
anecdotal data suggest that there are some critical issues of perception that need to be 
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addressed. One Clearinghouse Director noted that “Even the Department of Education does 
not seem to value or know about the comprehensive reach and service that ERIC provides. 
ERIC is perceived as a research tool for those with formal training” (Colker, 1999). Robert 
Stonehill (1992) identified several myths about ERIC that seem important to any future 
evaluation of ERIC; several have relevance to issues of future products and information 
dissemination. For example: ERIC is just a database; No one will buy documents from a 
publisher if they are available through ERIC; ERIC is not well-known to teachers or 
administrators, nor are ERIC products particularly useful to them. This last myth or 
misperception exemplifies the need for positive and effective marketing of ERIC to close the 
gap between myth and reality. The reality is, according to a New York Times report, that 
ERIC is the second most cited Internet site among Internet-using K-12 teachers. Further, the 
ESEA Prospectus (1999) indicated that Title 1 principals gave higher ratings to ERIC than to 
any other federal source of technical assistance. (Stubbs, 1999c). 

GOING FORWARD: BUILDING UPON ERICS STRENGTHS 

The quality of ERIC products has consistently resided in the accountability and 
quality assurance processes inherent in a strong participatory professional collaboration, 
including use of subject matter experts and peer reviews and collaboration. Although 
there is an awareness and recognition of the seriousness of ERIC’s weaknesses, obstacles 
and threats, there is an overwhelming consensus that ERIC’s strengths are the foundation of 
its longevity and are the key to its future. Some of the strengths identified by ERIC Directors 
and/or their staff in the Carruthers Survey are instructive in this regard. The following 
represent only a sampling: 

• Access to the ERIC database is available in a variety of formats 

• ERIC is an established service 

• ERIC’s reputation for quality 
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• ERIC’s talented staff; ERIC has experts in all the major areas education who 
are constantly exposed to the latest education literature in their areas 

• Dissemination via the Internet 

• The fact that most of ERIC’s information/products/services are free and 
instantly available on the Web. 

Subject Matter Experts 

A consistent theme in the data is that ERIC' s greatest internal asset is its people, 
staff, partners and contractors, and their abilities to be innovative and flexible. The 
decentralized structure and the availability of subject matter experts is the unique competitive 
edge ERIC has to meet the challenges for new product development and delivery 
mechanisms. The strengths of ERIC'S utilization of subject experts and its organizational 
partners are repeatedly affirmed by the data. 

Peer Collaboration 

As the earlier discussion of partnering suggested, at the heart of ERIC’s dissemination 
strategy is collaboration, both with peers and outside organizations. Of the $10 million 
budget, 78 percent is spent through its 16 subject-specific clearinghouses and 1 5 percent for 
publications. There are 13 adjunct clearinghouses and 2 affiliate clearinghouses. ERIC has 
acquisition arrangements with 2,190 research centers, universities, professional 
organizations, and federal and state agencies. In-kind contribution from host organizations is 
reported to be approximately 12 percent of the total ERIC budget. Further, ERIC links to the 
Web sites of 650 partner institutions and organizations that can distribute ERIC products. 
ERIC staffers participated in more the 470 education-related meetings, conferences, and 
workshops and gave more than 230 presentations in 1998. (Smarte, 1999). 

The ERIC Executive Committee monitors new and emerging issues and plans the 
national agenda. The ERIC Technical Steering Committee serves as a conduit between the 
ERIC system’s technical processing staff and ERIC Program Office. The ERIC Vocabulary 
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Review Group is responsible for the development and maintenance of ERIC's controlled 
vocabulary (ERIC Clearinghouse Statement of Work, p. 12). These committees are reported 
to have small budgets. It is possible that to further facilitate more efficiency with less, more 
support must be given to these types of system wide initiatives. 

Potential for Shared Product Development 

There are some new product ideas and issues that address the shared needs and 
interests of two or more clearinghouses. These items need to be identified for purposes of 
fostering greater collaboration and resource leveraging. For example, an online document 
to explain how to conduct, assess and interpret educational research and drug prevention 
education is a need expressed by several clearinghouses. New product development will be 
driven by new users to ERIC and a focus upon emergent and innovative materials 
development. To a large extent, a wealth of material relevant to present and future user 
requirements will not be in conventional bound publications. Given the growth in the 
international user community, there is also a need for translation of materials into other 
languages. ERIC needs to devise a workable plan for addressing this need. 

Cost of Success 

Because of the resource limitations, ERIC will have to pay special attention to 

managing increased marketing, technology increased demand and expectations, and 

demonstrated successes. There are clearinghouse experiences that suggest with increased 

advertising, major legislation and/or heated public policy debate inquiries to clearinghouses 

can be overwhelming. Several clearinghouses noted that they are unable to keep up with the 

demand for information. The 1987 OERI policy mandate pronounced: 

ERIC products and services should become more widely used 
and available; ERIC should become better integrated into 
OERIs mission of gathering, analyzing and reporting information 
on the status and condition of American education; and ERIC 
should serve a wider, more diverse audience, including policymakers, 
journalists, practitioners and the general public. (Goals 2000, p. 87) 
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If ERIC is to continue to be successful in meeting growing demands and rising 
expectations, issues of stakeholder alignment appear critical to the system’s future 
opportunity for product development and information dissemination. If opportunity is a 
prerequisite for success, ERIC has great possibility for the twenty-first century as it supports 
and links to the Department of Education priorities, goals and initiatives, particularly its 
Goals 2000. In spite of the various limitations it faces, ERIC is positioned to be a centerpiece 
of the National Library of Education. Blaine Dessey, Director of the National Library of 
Education puts it this way: 

ERIC is really our biggest asset in terms of what we are 
trying to do with the National Library... ERIC forums a 
huge hunk of our national education information agenda 
because it’s already done so much in terms of gaining a 
handle on the literature and making it available (quoted in 
Colker, 1999, p. 28). 

There is always risk associated with valuation of any asset--the risk of depreciation or 
the affirmation of appreciation. ERIC’s history warrants a full system wide evaluation, one 
very likely to prove its true value to American education over the past 30 plus years and chart 
its future value in the new millennium. 

CONSIDERATIONS FOR SYSTEMWIDE EVALUATION 

It is good to know, from anecdotal and other nonsystematic data, that a system is 
“working” and, in fact, “working very well.” Such data are not to be trivialized as they 
provide important dynamic data about a system. However, there is little doubt that more 
rigorous data collection in a well-designed evaluation schema is key to the planning of 
ERIC’s future and its response to the rising expectations of its clients. Contrary to the 
ongoing debate and contention between quantitative and qualitative researchers, the fact is 
that no method of data collection is inherently better than another one. Both quantitative and 
qualitative methods can facilitate the development of meaningful evaluation questions and 
yield significant information about a system. Increasingly evaluation researchers and others 
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have come to recognize the fact that we must consider multiple methodological approaches 
for the best results. Evaluation and other scientific research have shown clearly how 
combining methodologies can promote a clearer and more accurate understanding of that 
under investigation. It is within this framework that the following design and methodology is 
recommended. 

Recommended Design and Methodology 

The effectiveness and appropriateness of ERIC’s work products must be tied to a 
variety of user groups and utilize multiple methodologies. The foundational mission of ERIC 
as an archival and research database points to one discreet audience and a particular set of 
past, present and future needs. The mandate and mission of ERIC as a nerve center to foster 
and support the dynamic needs and interests of practitioners and stakeholders engaged in 
America’s educational reform may suggest another kind of audience for evaluating product 
effectiveness and appropriateness. Finally, justification for ERIC’s raison d’etre and funding 
level requests speaks to another cluster of evaluation issues. 

Given the major findings herein, it appears that evaluating the effectiveness and 
appropriateness of ERIC products and information dissemination processes center around 
five basic factors and benchmarks: (1) quality (2) access (3) depth (4) practicality, and (5) 
cost benefit. A balance between ERIC’s human resource capacity, technology utilization, 
and funding is the framework in which these questions must ultimately be addressed. 
Unquestionably, there is internal capacity and expertise within the ERIC community to 
undertake the kinds of evaluation activities required. However, there is also a dire need for 
external evaluative processes. 

Building upon the decentralized strengths of ERIC’s organization should be an 
integral part of the evaluation process. Given the dearth of systematic evaluation, the 
clearinghouses and partners are ERIC’s best resource and repositories for unearthing “what 
worked” and “what did not work.” A participatory process whereby they would be granted 
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resources to engage in a self-study to identify key products and information dissemination 
vehicles, past and present, as a precursor to external evaluation would be invaluable and is 
recommended. 

Designing the parameters for self-study, synthesizing and refining the results by an 
external team could be Phase One. The information provided would serve to shape the 
second phase, end user evaluations, to be undertaken by the external team. Phase two would 
have to address the multiple types of audiences ERIC already serves, as well as address the 
possibility of emergent and new audience needs. The dynamic nature of technology 
absolutely suggests an evaluation model that utilizes both quantitative and qualitative 
approaches in a traditional scientific sense. However, models more related to product 
marketing research and information management (e.g. data mining, trend studies, and focus 
groups) are also recommended. To ascertain how ERIC can best meet user demand and 
support public access to information and knowledge creation, some attention also must be 
given to an analysis of the competitive forces, private, profit and nonprofit, that are shaping 
the educational product development and dissemination landscape. And, given its history of 
limited funding, any evaluation of ERIC products must also focus specifically on what level 
of clearinghouse resources should be assigned to the development of new Website content 
and services. 

Suggested Key Questions 

Future systematic as well as clearinghouse specific evaluations (regardless of 
methodology) should be framed around a number of key questions. These key questions 
emerged from a systematic review of a variety of data about ERIC products and the future of 
product development relative to the rising expectations of old and new customers. The 
questions cluster around issues of product success, user needs, quality and standardization, 
and clearinghouses and their partners. In the final section of this paper is a listing of potential 
questions that might guide a future system wide and systematic evaluation of ERIC. After 
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each question is a number. The number indicates where the evaluation question fits relative 
to the five benchmarks: (1) quality, (2) access, (3) depth, (4) practicality, and (5) cost 
benefit. 



Product Success 



• What have been the types of publications disseminated by ERIC in the past 5 
10 Years? (2) 

• Which publications/products have proven most successful? (2) 

• What appears to have been the most effective means to get 
products/publications to ERIC audiences? (2) 

• What seems to be the most effective mix of print and electronic products to 
meet the needs of today’s ERIC audiences? (2,3) 

• In what subject areas or operational areas can there be clearinghouse overlap 
and opportunities for collaboration? (5) 

• What mechanisms are in place or can be instituted to give ongoing feedback 
about user needs and interests? (4) 

• How does the need for knowledge-building, related to holistic and life long 
learning educational approaches, impact ERIC’s future? (1,3) 

User Needs 



• Who are the past and present user communities of ERIC? How do you 
organize or categorize them as a defined targeted population? (2) 

• How do these users come to know about ERIC? (2) 

• What, if any, other educational database resources or services do ERIC 
customers use? (4) 

• What, if any, advantages does ERIC have over other educational databases or 
services? (1) 

• In general, what kinds of products/publications do users identify as most 
useful for them? (4) 

• Specifically, what kinds of print products/publications do users identify as 
most useful for them? (4) 

• Specifically, what kinds of Web-based or electronic products/publications do 
users identify as most useful to them? (4) 

• How do users gain access to ERIC products/publications? (2) 

• What other kinds of media (e g. video files) do ERIC users want? Can ERIC 
adequately provide such materials through collaboration with selected new 
partners? (3) 

Quality and Standardization 




What has been the long-term impact of low-level funding on ERIC’s ability to 
meet user needs and demands? (5) 

How can the quality of ERIC products be maintained with fewer resources? How 
does this impact what is expected from ERIC? (1) 

What standards can be established for purging, superseding or otherwise 
presenting some information in varying formats? (3) 
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• How is quality maintained or measured with so much focus upon numbers (e.g. 
the number of hits to a Web site, the number of publications distributed)? (1) 

• Are there common elements or attributes that should characterize every ERIC 
product and transaction? If so, what are they? (1) 

Clearinghouses and Partnerships 

• How can ERIC take a leadership role in offering real time conversations with 
subject matter experts or expert practitioners? (1) 

• Can ERIC maximize its reach by expanding its partnerships? (2) 

• Can ERIC increase its quality by limiting its scope? (2) 

• How are subject matter experts used at the clearinghouses? (3) 

• Should the role and use of subject matter experts be expanded and, if so, how 
should they be expanded? (3) 

• What is the utility of programs such as a Senior ERIC Associates for expanding 
the use of subject matter experts? (3) 

• How can user councils support ERIC’s ongoing understanding of user needs and 
hot topics? (4) 

• What system wide standards can clearinghouses adopt to establish greater 
consistency? (1) 
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